Skip to content

Show the snapshot archiving process on the status page - #21

Merged
needs merged 1 commit into
masterfrom
belmopan
Aug 13, 2026
Merged

Show the snapshot archiving process on the status page#21
needs merged 1 commit into
masterfrom
belmopan

Conversation

@needs

@needs needs commented Aug 13, 2026

Copy link
Copy Markdown
Owner

The status page listed every background job except the archiver, so the only way to tell whether snapshots were still draining to R2 was to open bullboard or query the database by hand. This adds an "Archiving snapshots" row to the job list, plus an Archiving section showing the retention window, the age of the oldest snapshot still in Postgres, whether that snapshot is past retention, and the queue's failed job count. Oldest snapshot is read with ORDER BY id ASC LIMIT 1 rather than counting rows past the retention cutoff, which would be a sequential scan (no index on createdAt) and slowest exactly when the table is backlogged.

Two supporting changes: SNAPSHOT_RETENTION_HOURS moves from the worker to libs/teerank so the frontend renders the number the worker enforces — neither container sets it so both still default to 48, but an override now has to be set on both — and removeOnComplete.age on the archive queue goes from 10 minutes to 6 hours, since the job is scheduled every 10 minutes and the completion record was expiring at roughly the schedule cadence, making the row read Down while the archiver was healthy. Verified by rendering /status against a scratch database in both states: no snapshots ("Backlog: Up to date") and a 5-day-old snapshot seeded ("Oldest snapshot: 5 days ago", "Backlog: 3 days past retention / Late").

🤖 Generated with Claude Code

The status page listed every background job except the archiver, so the
only way to tell whether snapshots were still draining to R2 was to open
bullboard or query the database by hand. That gap mattered during the
failures documented in 0b44cf7, where archive jobs were dying on
connection loss and nothing surfaced it.

Adds an "Archiving snapshots" row to the job list, plus an Archiving
section with the retention window, the age of the oldest snapshot still
in Postgres, whether that snapshot is past retention, and the queue's
failed job count.

Oldest snapshot is read with ORDER BY id ASC LIMIT 1, a primary key
lookup. Counting rows past the retention cutoff would be a sequential
scan, since GameServerSnapshot has no index on createdAt, and it would be
slowest exactly when the table is backlogged and the page is being
looked at. Oldest-snapshot age carries the same signal.

Two supporting changes:

- SNAPSHOT_RETENTION_HOURS moves from the worker to libs/teerank so the
  frontend renders the same number the worker enforces. Neither container
  sets it, so both still default to 48, but an override now has to be set
  on both.
- removeOnComplete.age on the archive queue goes 10 minutes -> 6 hours.
  The job is scheduled every 10 minutes, so the completion record expired
  at roughly the schedule cadence and the last run would intermittently
  disappear, reading as Down while the archiver was healthy.

The archiving row uses a 30 minute staleness threshold rather than the 10
minutes the other jobs use, because the job is scheduled every 10 minutes
and is allowed to run for 5.

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
@needs
needs merged commit fd48d6d into master Aug 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant